{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 循环神经网络\n",
    "\n",
    "本节将介绍如何在MNIST数据集中，搭建一个简单的循环神经网络，使用Tensorflow将进行手写字体分为0-9之间的10个类别。\n",
    "\n",
    "## 代码环境\n",
    "1. Python 3.6.1\n",
    "1. Tensorflow 1.4.0\n",
    "1. Jupyter 4.3.0\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- import 需要的库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.contrib import rnn\n",
    "from tensorflow.examples.tutorials.mnist import input_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- import MNIST data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data/train-images-idx3-ubyte.gz\n",
      "Extracting data/train-labels-idx1-ubyte.gz\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "mnist = input_data.read_data_sets(\"data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建模型\n",
    "首先设置训练的超参数，分别设置学习率、训练轮数和每轮训练的数据大小：\n",
    "- 设置训练的超参数，学习率为0.001、训练轮数100000次，以及batch_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lr = 0.001\n",
    "training_iters = 100000\n",
    "batch_size = 128\n",
    "display_step = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了使用RNN来做图片分类，可以把图片看成一个像素序列。MNIST图片的大小是28x28像素，所以把每一个图像样本看成一行行的序列。因此共有（28个元素序列）x（28行），每一步输入序列的长度是28，输入的步数是28步\n",
    "- 设置神经网络参数，序列长度28，步数28，隐藏单元128，分类的类别10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_input = 28\n",
    "n_step = 28\n",
    "n_hidden = 128\n",
    "n_classes = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义输入数据以及权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, [None, n_step, n_input])\n",
    "y = tf.placeholder(tf.float32, [None, n_classes])\n",
    "\n",
    "weights = {\n",
    "    # (28,128)\n",
    "    'in': tf.Variable(tf.random_normal([n_input,n_hidden])),\n",
    "    # (128,10)\n",
    "    'out': tf.Variable(tf.random_normal([n_hidden,n_classes]))\n",
    "}\n",
    "\n",
    "biases = {\n",
    "    # (128)\n",
    "    'in': tf.Variable(tf.constant(0.1,shape=[n_hidden,])),\n",
    "    # (10,)\n",
    "    'out': tf.Variable(tf.constant(0.1,shape=[n_classes,]))\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义RNN模型\n",
    "- 把输入的X转换成　X ==> (128 batch*28 steps,28 inputs)\n",
    "- 采用基本的LSTM循环网络单元\n",
    "- 输出该序列的各个分类概率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def rnn_model(X,weights,biases):\n",
    "    # X ==> (128 batch*28 steps,28 inputs)\n",
    "    X = tf.reshape(X,[-1,n_input])\n",
    "    # X_in = (128 batch*28 steps,128 hidden)\n",
    "    X_in = tf.matmul(X,weights['in']+biases['in'])\n",
    "    # X_in ==> (128 batch,28 steps,128 hidden)\n",
    "    X_in = tf.reshape(X_in,[-1,n_step,n_hidden])\n",
    "\n",
    "    #use basic LSTM Cell\n",
    "    lstm_cell = rnn.BasicLSTMCell(n_hidden,forget_bias=1.0,\n",
    "                                  state_is_tuple=True)\n",
    "    init_state = lstm_cell.zero_state(batch_size,dtype=tf.float32)\n",
    "    outputs,final_state = tf.nn.dynamic_rnn(lstm_cell,X_in,initial_state=init_state,\n",
    "                                            time_major=False)\n",
    "    results = tf.matmul(final_state[1],weights['out'] + biases['out'])\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义损失函数和优化器，优化器采用AdamOptimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pred = rnn_model(x,weights,biases)\n",
    "cost = tf.reduce_mean(\n",
    "    tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))\n",
    "optimizer = tf.train.AdamOptimizer(lr).minimize(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义模型预测结果以及准确率的计算方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "correct_pred = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练数据以及评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1280, Minibatch Loss= 1.301216, Training Accuracy= 0.55469\n",
      "Iteration 2560, Minibatch Loss= 1.010427, Training Accuracy= 0.66406\n",
      "Iteration 3840, Minibatch Loss= 0.788430, Training Accuracy= 0.70312\n",
      "Iteration 5120, Minibatch Loss= 0.667551, Training Accuracy= 0.78906\n",
      "Iteration 6400, Minibatch Loss= 0.595816, Training Accuracy= 0.78125\n",
      "Iteration 7680, Minibatch Loss= 0.350965, Training Accuracy= 0.88281\n",
      "Iteration 8960, Minibatch Loss= 0.499101, Training Accuracy= 0.79688\n",
      "Iteration 10240, Minibatch Loss= 0.482088, Training Accuracy= 0.82812\n",
      "Iteration 11520, Minibatch Loss= 0.504503, Training Accuracy= 0.83594\n",
      "Iteration 12800, Minibatch Loss= 0.271221, Training Accuracy= 0.91406\n",
      "Iteration 14080, Minibatch Loss= 0.464995, Training Accuracy= 0.86719\n",
      "Iteration 15360, Minibatch Loss= 0.322582, Training Accuracy= 0.89844\n",
      "Iteration 16640, Minibatch Loss= 0.347899, Training Accuracy= 0.88281\n",
      "Iteration 17920, Minibatch Loss= 0.394192, Training Accuracy= 0.88281\n",
      "Iteration 19200, Minibatch Loss= 0.213484, Training Accuracy= 0.94531\n",
      "Iteration 20480, Minibatch Loss= 0.294130, Training Accuracy= 0.92188\n",
      "Iteration 21760, Minibatch Loss= 0.258474, Training Accuracy= 0.92969\n",
      "Iteration 23040, Minibatch Loss= 0.385059, Training Accuracy= 0.89062\n",
      "Iteration 24320, Minibatch Loss= 0.264384, Training Accuracy= 0.89844\n",
      "Iteration 25600, Minibatch Loss= 0.288544, Training Accuracy= 0.92969\n",
      "Iteration 26880, Minibatch Loss= 0.439751, Training Accuracy= 0.86719\n",
      "Iteration 28160, Minibatch Loss= 0.200464, Training Accuracy= 0.95312\n",
      "Iteration 29440, Minibatch Loss= 0.362614, Training Accuracy= 0.87500\n",
      "Iteration 30720, Minibatch Loss= 0.279403, Training Accuracy= 0.92188\n",
      "Iteration 32000, Minibatch Loss= 0.221065, Training Accuracy= 0.93750\n",
      "Iteration 33280, Minibatch Loss= 0.254707, Training Accuracy= 0.91406\n",
      "Iteration 34560, Minibatch Loss= 0.186574, Training Accuracy= 0.96094\n",
      "Iteration 35840, Minibatch Loss= 0.209275, Training Accuracy= 0.93750\n",
      "Iteration 37120, Minibatch Loss= 0.200519, Training Accuracy= 0.92188\n",
      "Iteration 38400, Minibatch Loss= 0.160687, Training Accuracy= 0.94531\n",
      "Iteration 39680, Minibatch Loss= 0.298483, Training Accuracy= 0.91406\n",
      "Iteration 40960, Minibatch Loss= 0.201895, Training Accuracy= 0.92188\n",
      "Iteration 42240, Minibatch Loss= 0.158606, Training Accuracy= 0.94531\n",
      "Iteration 43520, Minibatch Loss= 0.307986, Training Accuracy= 0.90625\n",
      "Iteration 44800, Minibatch Loss= 0.281966, Training Accuracy= 0.88281\n",
      "Iteration 46080, Minibatch Loss= 0.261283, Training Accuracy= 0.89844\n",
      "Iteration 47360, Minibatch Loss= 0.291441, Training Accuracy= 0.91406\n",
      "Iteration 48640, Minibatch Loss= 0.202818, Training Accuracy= 0.92188\n",
      "Iteration 49920, Minibatch Loss= 0.113422, Training Accuracy= 0.96875\n",
      "Iteration 51200, Minibatch Loss= 0.105692, Training Accuracy= 0.96875\n",
      "Iteration 52480, Minibatch Loss= 0.154081, Training Accuracy= 0.96094\n",
      "Iteration 53760, Minibatch Loss= 0.145414, Training Accuracy= 0.95312\n",
      "Iteration 55040, Minibatch Loss= 0.117242, Training Accuracy= 0.96094\n",
      "Iteration 56320, Minibatch Loss= 0.081149, Training Accuracy= 0.97656\n",
      "Iteration 57600, Minibatch Loss= 0.108463, Training Accuracy= 0.95312\n",
      "Iteration 58880, Minibatch Loss= 0.156470, Training Accuracy= 0.96094\n",
      "Iteration 60160, Minibatch Loss= 0.148587, Training Accuracy= 0.95312\n",
      "Iteration 61440, Minibatch Loss= 0.237871, Training Accuracy= 0.92969\n",
      "Iteration 62720, Minibatch Loss= 0.147145, Training Accuracy= 0.96094\n",
      "Iteration 64000, Minibatch Loss= 0.098019, Training Accuracy= 0.96875\n",
      "Iteration 65280, Minibatch Loss= 0.118203, Training Accuracy= 0.96094\n",
      "Iteration 66560, Minibatch Loss= 0.101285, Training Accuracy= 0.96875\n",
      "Iteration 67840, Minibatch Loss= 0.207359, Training Accuracy= 0.93750\n",
      "Iteration 69120, Minibatch Loss= 0.067886, Training Accuracy= 0.97656\n",
      "Iteration 70400, Minibatch Loss= 0.161458, Training Accuracy= 0.93750\n",
      "Iteration 71680, Minibatch Loss= 0.138106, Training Accuracy= 0.96094\n",
      "Iteration 72960, Minibatch Loss= 0.073405, Training Accuracy= 0.98438\n",
      "Iteration 74240, Minibatch Loss= 0.143483, Training Accuracy= 0.96094\n",
      "Iteration 75520, Minibatch Loss= 0.097661, Training Accuracy= 0.97656\n",
      "Iteration 76800, Minibatch Loss= 0.118980, Training Accuracy= 0.95312\n",
      "Iteration 78080, Minibatch Loss= 0.124437, Training Accuracy= 0.97656\n",
      "Iteration 79360, Minibatch Loss= 0.128721, Training Accuracy= 0.95312\n",
      "Iteration 80640, Minibatch Loss= 0.162701, Training Accuracy= 0.95312\n",
      "Iteration 81920, Minibatch Loss= 0.070164, Training Accuracy= 0.98438\n",
      "Iteration 83200, Minibatch Loss= 0.077578, Training Accuracy= 0.98438\n",
      "Iteration 84480, Minibatch Loss= 0.138588, Training Accuracy= 0.96094\n",
      "Iteration 85760, Minibatch Loss= 0.162362, Training Accuracy= 0.95312\n",
      "Iteration 87040, Minibatch Loss= 0.135977, Training Accuracy= 0.94531\n",
      "Iteration 88320, Minibatch Loss= 0.129117, Training Accuracy= 0.96094\n",
      "Iteration 89600, Minibatch Loss= 0.148080, Training Accuracy= 0.95312\n",
      "Iteration 90880, Minibatch Loss= 0.122423, Training Accuracy= 0.96875\n",
      "Iteration 92160, Minibatch Loss= 0.207287, Training Accuracy= 0.94531\n",
      "Iteration 93440, Minibatch Loss= 0.246922, Training Accuracy= 0.93750\n",
      "Iteration 94720, Minibatch Loss= 0.140132, Training Accuracy= 0.93750\n",
      "Iteration 96000, Minibatch Loss= 0.063141, Training Accuracy= 0.97656\n",
      "Iteration 97280, Minibatch Loss= 0.036757, Training Accuracy= 0.99219\n",
      "Iteration 98560, Minibatch Loss= 0.062806, Training Accuracy= 0.97656\n",
      "Iteration 99840, Minibatch Loss= 0.107706, Training Accuracy= 0.96875\n",
      "Optimization Finished!\n"
     ]
    }
   ],
   "source": [
    "tf.summary.scalar('accuracy', accuracy)\n",
    "tf.summary.scalar('loss', cost)\n",
    "summaries = tf.summary.merge_all()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    train_writer = tf.summary.FileWriter('logs/', sess.graph)\n",
    "    init = tf.global_variables_initializer()\n",
    "    sess.run(init)\n",
    "    step = 1\n",
    "    while batch_size * step < training_iters:\n",
    "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
    "        batch_x = batch_x.reshape(batch_size, n_step, n_input)\n",
    "        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})\n",
    "        if step % display_step == 0:\n",
    "            acc, loss = sess.run(\n",
    "                [accuracy, cost], feed_dict={x: batch_x,\n",
    "                                             y: batch_y})\n",
    "            print(\"Iteration \" + str(step * batch_size) + \", Minibatch Loss= \" + \\\n",
    "                  \"{:.6f}\".format(loss) + \", Training Accuracy= \" + \\\n",
    "                  \"{:.5f}\".format(acc))\n",
    "        if step % 100 == 0:\n",
    "            s = sess.run(summaries, feed_dict={x: batch_x, y: batch_y})\n",
    "            train_writer.add_summary(s, global_step=step)\n",
    "\n",
    "        step += 1\n",
    "    print(\"Optimization Finished!\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "代码参考[https://github.com/nlintz/TensorFlow-Tutorials/blob/master/07_lstm.py](https://github.com/nlintz/TensorFlow-Tutorials/blob/master/07_lstm.py \"title\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
